Evaluation Metrics for Automatically Generated Metaphorical Expressions
نویسندگان
چکیده
This paper proposesmetrics to evaluate the quality of automatically generatedmetaphors not restricted to similes. The metrics are metaphoricity, novelty, comprehensibility, and overall evaluation. First, we discuss their importance and necessity. Next, we show that it is feasible to evaluate them by crowdsourcing. The targets of the evaluation are 1,360 expressions, each of which consists of a noun taken from a list of 40 nouns and a verbal phrase taken from a list of 34 verbal phrases. Then, we analyze the results to check the validity of the metaphoricity, novelty, and comprehensibility, and clarify their relationship. Finally, we argue that high-ranked expressions in the overall evaluation are considered to be “good metaphors” by showing that they actually are metaphors and are preferred by a human judge.
منابع مشابه
Automated error analysis for multiword expressions: Using BLEU-type scores for automatic discovery of potential translation errors
We describe the results of a research project aimed at automatic detection of MT errors using state-of-the-art MT evaluation metrics, such as BLEU. Currently, these automated metrics give only a general indication of translation quality at the corpus level and cannot be used directly for identifying gaps in the coverage of MT systems. Our methodology uses automatic detection of frequent multiwo...
متن کاملAutomatic Metaphor Interpretation as a Paraphrasing Task
We present a novel approach to metaphor interpretation and a system that produces literal paraphrases for metaphorical expressions. Such a representation is directly transferable to other applications that can benefit from a metaphor processing component. Our method is distinguished from the previous work in that it does not rely on any hand-crafted knowledge about metaphor, but in contrast emp...
متن کاملCognitive Therapy Protocol of Understanding Embodiment Metaphorical Expressions in Children with William’s Syndrome
Background: Normal children understand and use embodiment metaphorical expressions since they start learning a language, but children suffering from William’s syndrome even in adulthood have little understanding of such expressions and they can hardly use them. Objectives: This study is an attempt to teach embodiment metaphorical expressions of 4.5-5 year old Persian children suffering...
متن کاملReading Times Predict the Quality of Generated Text Above and Beyond Human Ratings
Typically, human evaluation of NLG output is based on user ratings. We collected ratings and reading time data in a simple, low-cost experimental paradigm for text generation. Participants were presented corpus texts, automatically linearised texts, and texts containing predicted referring expressions and automatic linearisation. We demonstrate that the reading time metrics outperform the ratin...
متن کاملDetecting Metaphorical Phrases in the Polish Language
In this paper we describe experiments with automated detection of metaphors in the Polish language. We focus our analysis on noun phrases composed of an adjective and a noun, and distinguish three types of expressions: with literal sense, with metaphorical sense, and expressions both literal and methaphorical (context-dependent). We propose a method of automatically recognizing expression type ...
متن کامل